Optimizing an adaptive digital oral health intervention for promoting oral self-care behaviors: Micro-randomized trial protocol.

Dental disease continues to be one of the most prevalent chronic diseases in the United States. Although oral self-care behaviors (OSCB), involving systematic twice-a-day tooth brushing, can prevent dental disease, this basic behavior is not sufficiently practiced. Recent advances in digital technology offer tremendous potential for promoting OSCB by delivering Just-In-Time Adaptive Interventions (JITAIs)- interventions that leverage dynamic information about the person's state and context to effectively prompt them to engage in a desired behavior in real-time, real-world settings. However, limited research attention has been given to systematically investigating how to best prompt individuals to engage in OSCB in daily life, and under what conditions prompting would be most beneficial. This paper describes the protocol for a Micro-Randomized Trial (MRT) to inform the development of a JITAI for promoting ideal OSCB, namely, brushing twice daily, for two minutes each time, in all four dental quadrants (i.e., 2x2x4). Sensors within an electric toothbrush (eBrush) will be used to track OSCB and a matching mobile app (Oralytics) will deliver on-demand feedback and educational information. The MRT will micro-randomize participants twice daily (morning and evening) to either (a) a prompt (push notification) containing one of several theoretically grounded engagement strategies or (b) no prompt. The goal is to investigate whether, what type of, and under what conditions prompting increases engagement in ideal OSCB. The results will build the empirical foundation necessary to develop an optimized JITAI that will be evaluated relative to a suitable control in a future randomized controlled trial.

Intervention Optimization: A Paradigm Shift and Its Potential Implications for Clinical Psychology.

To build a coherent knowledge base about what psychological intervention strategies work, develop interventions that have positive societal impact, and maintain and increase this impact over time, it is necessary to replace the classical treatment package research paradigm. The multiphase optimization strategy (MOST) is an alternative paradigm that integrates ideas from behavioral science, engineering, implementation science, economics, and decision science. MOST enables optimization of interventions to strategically balance effectiveness, affordability, scalability, and efficiency. In this review we provide an overview of MOST, discuss several experimental designs that can be used in intervention optimization, consider how the investigator can use experimental results to select components for inclusion in the optimized intervention, discuss the application of MOST in implementation science, and list future issues in this rapidly evolving field. We highlight the feasibility of adopting this new research paradigm as well as its potential to hasten the progress of psychological intervention science. Expected final online publication date for the Annual Review of Clinical Psychology, Volume 20 is May 2024. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Developing Message Strategies to Engage Racial and Ethnic Minority Groups in Digital Oral Self-Care Interventions: Participatory Co-Design Approach.

BACKGROUND: The prevention of oral health diseases is a key public health issue and a major challenge for racial and ethnic minority groups, who often face barriers in accessing dental care. Daily toothbrushing is an important self-care behavior necessary for sustaining good oral health, yet engagement in regular brushing remains a challenge. Identifying strategies to promote engagement in regular oral self-care behaviors among populations at risk of poor oral health is critical. OBJECTIVE: The formative research described here focused on creating messages for a digital oral self-care intervention targeting a racially and ethnically diverse population. Theoretically grounded strategies (reciprocity, reciprocity-by-proxy, and curiosity) were used to promote engagement in 3 aspects: oral self-care behaviors, an oral care smartphone app, and digital messages. A web-based participatory co-design approach was used to develop messages that are resource efficient, appealing, and novel; this approach involved dental experts, individuals from the general population, and individuals from the target population-dental patients from predominantly low-income racial and ethnic minority groups. Given that many individuals from racially and ethnically diverse populations face anonymity and confidentiality concerns when participating in research, we used an approach to message development that aimed to mitigate these concerns. METHODS: Messages were initially developed with feedback from dental experts and Amazon Mechanical Turk workers. Dental patients were then recruited for 2 facilitator-mediated group webinar sessions held over Zoom (Zoom Video Communications; session 1: n=13; session 2: n=7), in which they provided both quantitative ratings and qualitative feedback on the messages. Participants interacted with the facilitator through Zoom polls and a chat window that was anonymous to other participants. Participants did not directly interact with each other, and the facilitator mediated sessions by verbally asking for message feedback and sharing key suggestions with the group for additional feedback. This approach plausibly enhanced participant anonymity and confidentiality during the sessions. RESULTS: Participants rated messages highly in terms of liking (overall rating: mean 2.63, SD 0.58; reciprocity: mean 2.65, SD 0.52; reciprocity-by-proxy: mean 2.58, SD 0.53; curiosity involving interactive oral health questions and answers: mean 2.45, SD 0.69; curiosity involving tailored brushing feedback: mean 2.77, SD 0.48) on a scale ranging from 1 (do not like it) to 3 (like it). Qualitative feedback indicated that the participants preferred messages that were straightforward, enthusiastic, conversational, relatable, and authentic. CONCLUSIONS: This formative research has the potential to guide the design of messages for future digital health behavioral interventions targeting individuals from diverse racial and ethnic populations. Insights emphasize the importance of identifying key stimuli and tasks that require engagement, gathering multiple perspectives during message development, and using new approaches for collecting both quantitative and qualitative data while mitigating anonymity and confidentiality concerns.

Assessing the Impact of Context Inference Error and Partial Observability on RL Methods for Just-In-Time Adaptive Interventions.

Just-in-Time Adaptive Interventions (JITAIs) are a class of personalized health interventions developed within the behavioral science community. JITAIs aim to provide the right type and amount of support by iteratively selecting a sequence of intervention options from a pre-defined set of components in response to each individual's time varying state. In this work, we explore the application of reinforcement learning methods to the problem of learning intervention option selection policies. We study the effect of context inference error and partial observability on the ability to learn effective policies. Our results show that the propagation of uncertainty from context inferences is critical to improving intervention efficacy as context uncertainty increases, while policy gradient algorithms can provide remarkable robustness to partially observed behavioral state information.

The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning.

Discount regularization, using a shorter planning horizon when calculating the optimal policy, is a popular choice to restrict planning to a less complex set of policies when estimating an MDP from sparse or noisy data (Jiang et al., 2015). It is commonly understood that discount regularization functions by de-emphasizing or ignoring delayed effects. In this paper, we reveal an alternate view of discount regularization that exposes unintended consequences. We demonstrate that planning under a lower discount factor produces an identical optimal policy to planning using any prior on the transition matrix that has the same distribution for all states and actions. In fact, it functions like a prior with stronger regularization on state-action pairs with more transition data. This leads to poor performance when the transition matrix is estimated from data sets with uneven amounts of data across state-action pairs. Our equivalence theorem leads to an explicit formula to set regularization parameters locally for individual state-action pairs rather than globally. We demonstrate the failures of discount regularization and how we remedy them using our state-action-specific method across simple empirical examples as well as a medical cancer simulator.

Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care.

While dental disease is largely preventable, professional advice on optimal oral hygiene practices is often forgotten or abandoned by patients. Therefore patients may benefit from timely and personalized encouragement to engage in oral self-care behaviors. In this paper, we develop an online reinforcement learning (RL) algorithm for use in optimizing the delivery of mobile-based prompts to encourage oral hygiene behaviors. One of the main challenges in developing such an algorithm is ensuring that the algorithm considers the impact of current actions on the effectiveness of future actions (i.e., delayed effects), especially when the algorithm has been designed to run stably and autonomously in a constrained, real-world setting characterized by highly noisy, sparse data. We address this challenge by designing a quality reward that maximizes the desired health outcome (i.e., high-quality brushing) while minimizing user burden. We also highlight a procedure for optimizing the hyperparameters of the reward by building a simulation environment test bed and evaluating candidates using the test bed. The RL algorithm discussed in this paper will be deployed in Oralytics. To the best of our knowledge, Oralytics is the first mobile health study utilizing an RL algorithm designed to prevent dental disease by optimizing the delivery of motivational messages supporting oral self-care behaviors.

Sample size considerations for micro-randomized trials with binary proximal outcomes.

Micro-randomized trials (MRTs) are a novel experimental design for developing mobile health interventions. Participants are repeatedly randomized in an MRT, resulting in longitudinal data with time-varying treatments. Causal excursion effects are the main quantities of interest in MRT primary and secondary analyses. We consider MRTs where the proximal outcome is binary and the randomization probability is constant or time-varying but not data-dependent. We develop a sample size formula for detecting a nonzero marginal excursion effect. We prove that the formula guarantees power under a set of working assumptions. We demonstrate via simulation that violations of certain working assumptions do not affect the power, and for those that do, we point out the direction in which the power changes. We then propose practical guidelines for using the sample size formula. As an illustration, the formula is used to size an MRT on interventions for excessive drinking. The sample size calculator is implemented in R package MRTSampleSizeBinary and an interactive R Shiny app. This work can be used in trial planning for a wide range of MRTs with binary proximal outcomes.

The engagement problem: A review of engagement with digital mental health interventions and recommendations for a path forward.

Purpose of the review: Digital mental health interventions (DMHIs) are an effective and accessible means of addressing the unprecedented levels of mental illness worldwide. Currently, however, patient engagement with DMHIs in real-world settings is often insufficient to see clinical benefit. In order to realize the potential of DMHIs, there is a need to better understand what drives patient engagement. Recent findings: We discuss takeaways from the existing literature related to patient engagement with DMHIs and highlight gaps to be addressed through further research. Findings suggest that engagement is influenced by patient-, intervention- and systems-level factors. At the patient-level, variables such as sex, education, personality traits, race, ethnicity, age and symptom severity appear to be associated with engagement. At the intervention-level, integrating human support, gamification, financial incentives and persuasive technology features may improve engagement. Finally, although systems-level factors have not been widely explored, the existing evidence suggests that achieving engagement will require addressing organizational and social barriers and drawing on the field of implementation science. Summary: Future research clarifying the patient-, intervention- and systems-level factors that drive engagement will be essential. Additionally, to facilitate improved understanding of DMHI engagement, we propose the following: (a) widespread adoption of a minimum necessary 5-element engagement reporting framework; (b) broader application of alternative clinical trial designs; and (c) directed efforts to build upon an initial parsimonious conceptual model of DMHI engagement.

Just-in-Time Adaptive Interventions for Suicide Prevention: Promise, Challenges, and Future Directions.

The suicide rate (currently 14 per 100,000) has barely changed in the United States over the past 100 years. There is a need for new ways of preventing suicide. Further, research has revealed that suicidal thoughts and behaviors and the factors that drive them are dynamic, heterogeneous, and interactive. Most existing interventions for suicidal thoughts and behaviors are infrequent, not accessible when most needed, and not systematically tailored to the person using their own data (e.g., from their own smartphone). Advances in technology offer an opportunity to develop new interventions that may better match the dynamic, heterogeneous, and interactive nature of suicidal thoughts and behaviors. Just-In-Time Adaptive Interventions (JITAIs), which use smartphones and wearables, are designed to provide the right type of support at the right time by adapting to changes in internal states and external contexts, offering a promising pathway toward more effective suicide prevention. In this review, we highlight the potential of JITAIs for suicide prevention, challenges ahead (e.g., measurement, ethics), and possible solutions to these challenges.

Cost and Effort Considerations for the Development of Intervention Studies Using Mobile Health Platforms: Pragmatic Case Study.

BACKGROUND: The research marketplace has seen a flood of open-source or commercial mobile health (mHealth) platforms that can collect and use user data in real time. However, there is a lack of practical literature on how these platforms are developed, integrated into study designs, and adopted, including important information around cost and effort considerations. OBJECTIVE: We intend to build critical literacy in the clinician-researcher readership into the cost, effort, and processes involved in developing and operationalizing an mHealth platform, focusing on Intui, an mHealth platform that we developed. METHODS: We describe the development of the Intui mHealth platform and general principles of its operationalization across sites. RESULTS: We provide a worked example in the form of a case study. Intui was operationalized in the design of a behavioral activation intervention in collaboration with a mental health service provider. We describe the design specifications of the study site, the developed software, and the cost and effort required to build the final product. CONCLUSIONS: Study designs, researcher needs, and technical considerations can impact effort and costs associated with the use of mHealth platforms. Greater transparency from platform developers about the impact of these factors on practical considerations relevant to end users such as clinician-researchers is crucial to increasing critical literacy around mHealth, thereby aiding in the widespread use of these potentially beneficial technologies and building clinician confidence in these tools.

Engagement in digital interventions.

The notion of "engagement," which plays an important role in various domains of psychology, is gaining increased currency as a concept that is critical to the success of digital interventions. However, engagement remains an ill-defined construct, with different fields generating their own domain-specific definitions. Moreover, given that digital interactions in real-world settings are characterized by multiple demands and choice alternatives competing for an individual's effort and attention, they involve fast and often impulsive decision-making. Prior research seeking to uncover the mechanisms underlying engagement has nonetheless focused mainly on psychological factors and social influences and neglected to account for the role of neural mechanisms that shape individual choices. This article aims to integrate theories and empirical evidence across multiple domains to define engagement and discuss opportunities and challenges to promote effective engagement in digital interventions. We also propose the affect-integration-motivation and attention-context-translation (AIM-ACT) framework, which is based on a neurophysiological account of engagement, to shed new light on how in-the-moment engagement unfolds in response to a digital stimulus. Building on this framework, we provide recommendations for designing strategies to promote engagement in digital interventions and highlight directions for future research. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The microrandomized trial for developing digital interventions: Experimental design and data analysis considerations.

Just-in-time adaptive interventions (JITAIs) are time-varying adaptive interventions that use frequent opportunities for the intervention to be adapted-weekly, daily, or even many times a day. The microrandomized trial (MRT) has emerged for use in informing the construction of JITAIs. MRTs can be used to address research questions about whether and under what circumstances JITAI components are effective, with the ultimate objective of developing effective and efficient JITAI. The purpose of this article is to clarify why, when, and how to use MRTs; to highlight elements that must be considered when designing and implementing an MRT; and to review primary and secondary analyses methods for MRTs. We briefly review key elements of JITAIs and discuss a variety of considerations that go into planning and designing an MRT. We provide a definition of causal excursion effects suitable for use in primary and secondary analyses of MRT data to inform JITAI development. We review the weighted and centered least-squares (WCLS) estimator which provides consistent causal excursion effect estimators from MRT data. We describe how the WCLS estimator along with associated test statistics can be obtained using standard statistical software such as R (R Core Team, 2019). Throughout we illustrate the MRT design and analyses using the HeartSteps MRT, for developing a JITAI to increase physical activity among sedentary individuals. We supplement the HeartSteps MRT with two other MRTs, SARA and BariFit, each of which highlights different research questions that can be addressed using the MRT and experimental design considerations that might arise. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

BATCH POLICY LEARNING IN AVERAGE REWARD MARKOV DECISION PROCESSES.

We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.

Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines.

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (predictability, computability, stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning to the design of RL algorithms for the digital interventions setting. Furthermore, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We show how we used the PCS framework to design an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

Translating strategies for promoting engagement in mobile health: A proof-of-concept microrandomized trial.

OBJECTIVE: Mobile technologies allow for accessible and cost-effective health monitoring and intervention delivery. Despite these advantages, mobile health (mHealth) engagement is often insufficient. While monetary incentives may increase engagement, they can backfire, dampening intrinsic motivations and undermining intervention scalability. Theories from psychology and behavioral economics suggest useful nonmonetary strategies for promoting engagement; however, examinations of the applicability of these strategies to mHealth engagement are lacking. This proof-of-concept study evaluates the translation of theoretically-grounded engagement strategies into mHealth, by testing their potential utility in promoting daily self-reporting. METHOD: A microrandomized trial (MRT) was conducted with adolescents and emerging adults with past-month substance use. Participants were randomized multiple times daily to receive theoretically-grounded strategies, namely reciprocity (the delivery of inspirational quote prior to self-reporting window) and nonmonetary reinforcers (e.g., the delivery of meme/gif following self-reporting completion) to improve proximal engagement in daily mHealth self-reporting. RESULTS: Daily self-reporting rates (62.3%; n = 68) were slightly lower than prior literature, albeit with much lower financial incentives. The utility of specific strategies was found to depend on contextual factors pertaining to the individual's receptivity and risk for disengagement. For example, the effect of reciprocity significantly varied depending on whether this strategy was employed (vs. not employed) during the weekend. The nonmonetary reinforcement strategy resulted in different outcomes when operationalized in various ways. CONCLUSIONS: While the results support the translation of the reciprocity strategy into this mHealth setting, the translation of nonmonetary reinforcement requires further consideration prior to inclusion in a full scale MRT. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Estimating time-varying causal excursion effect in mobile health with binary outcomes.

Advances in wearables and digital technology now make it possible to deliver behavioral mobile health interventions to individuals in their everyday life. The micro-randomized trial is increasingly used to provide data to inform the construction of these interventions. In a micro-randomized trial, each individual is repeatedly randomized among multiple intervention options, often hundreds or even thousands of times, over the course of the trial. This work is motivated by multiple micro-randomized trials that have been conducted or are currently in the field, in which the primary outcome is a longitudinal binary outcome. The primary aim of such micro-randomized trials is to examine whether a particular time-varying intervention has an effect on the longitudinal binary outcome, often marginally over all but a small subset of the individual's data. We propose the definition of causal excursion effect that can be used in such primary aim analysis for micro-randomized trials with binary outcomes. Under rather restrictive assumptions one can, based on existing literature, derive a semiparametric, locally efficient estimator of the causal effect. Starting from this estimator, we develop an estimator that can be used as the basis of a primary aim analysis under more plausible assumptions. Simulation studies are conducted to compare the estimators. We illustrate the developed methods using data from the micro-randomized trial, BariFit. In BariFit, the goal is to support weight maintenance for individuals who received bariatric surgery.

Understanding Adolescent and Young Adult 6-Mercaptopurine Adherence and mHealth Engagement During Cancer Treatment: Protocol for Ecological Momentary Assessment.

BACKGROUND: Adolescents and young adults (AYAs) with cancer demonstrate suboptimal oral chemotherapy adherence, increasing their risk of cancer relapse. It is unclear how everyday time-varying contextual factors (eg, mood) affect their adherence, stalling the development of personalized mobile health (mHealth) interventions. Poor engagement is also a challenge across mHealth trials; an effective adherence intervention must be engaging to promote uptake. OBJECTIVE: This protocol aims to determine the temporal associations between daily contextual factors and 6-mercaptopurine (6-MP) adherence and explore the proximal impact of various engagement strategies on ecological momentary assessment survey completion. METHODS: At the Children's Hospital of Philadelphia, AYAs with acute lymphoblastic leukemia or lymphoma who are prescribed prolonged maintenance chemotherapy that includes daily oral 6-MP are eligible, along with their matched caregivers. Participants will use an ecological momentary assessment app called ADAPTS (Adherence Assessments and Personalized Timely Support)-a version of an open-source app that was modified for AYAs with cancer through a user-centered process-and complete surveys in bursts over 6 months. Theory-informed engagement strategies will be microrandomized to estimate the causal effects on proximal survey completion. RESULTS: With funding from the National Cancer Institute and institutional review board approval, of the proposed 30 AYA-caregiver dyads, 60% (18/30) have been enrolled; of the 18 enrolled, 15 (83%) have completed the study so far. CONCLUSIONS: This protocol represents an important first step toward prescreening tailoring variables and engagement components for a just-in-time adaptive intervention designed to promote both 6-MP adherence and mHealth engagement. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/32789.